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DETAILED ACTION 

1 . Claims 1 - 29 are pending. 

Information Disclosure Statement 

2. The information disclosure statement filed May 26, 2006 fails to comply with the 
provisions of 37 CFR 1.97, 1.98 and MPEP § 609 because reference "Named faces: Putting 
Names to Faces" isn't provided. It has been placed in the application file, but the information 
referred to therein has not been considered as to the merits. Applicant is advised that the date of 
any re-submission of any item of information contained in this information disclosure statement 
or the submission of any missing element(s) will be the date of submission for purposes of 
determining compliance with the requirements based on the time of filing the statement, 
including all certification requirements for statements under 37 CFR 1.97(e). See MPEP 

§ 609.05(a). 

3. All other IDS filed on May 26, 2006 has considered by the Examiner. 

Oath/Declaration 

4. The full name of each inventor (family name and at least one given name together with 
any initial) has not been set forth. The listed inventor Turetsky, Robert is not disclosed in the 
Oath. 

Claim Rejections - 35 USC § 101 

5. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 
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6. Claims 1- 25 are rejected under 35 U.S.C. 101 because the claimed invention is directed 
to non-statutory subject matter. 

7. Claims 1-25 discloses a system for intergrative analysis of intrinsic and extrinsic audio- 
visual data but fails to disclose a processor and a computer readable medium therefor lacks the 
necessary physical articles or objects to constitute a machine or a manufacture within the 
meaning of 35 USC 101 . They are clearly not a series of steps or acts to be a process nor are they 
a combination of chemical compounds to be a composition of matter. As such, they fail to fall 
within a statutory category. They are, at best, functional descriptive material per se. 

Claim Rejections - 35 USC § 102 

8. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

9. Claims 1-5, 8-9, 12, 17-23, 26, 29 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Ikezoye et al. (US 6834308 Bl). 

As per claim 1, Ikezoye discloses: 

• A system (100) for integrative analysis of intrinsic (10), as (col. 2 lines 51-52) 
"generates a media sample or analytical representation of the media content", and 
extrinsic (II) audio-visual data, as (col. 2 lines 53 - 55) "media sample or 
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representation is compared to a database of the sampled media content or representations 
to query and ascertain information related to the sample", the system comprising: 

• an intrinsic content analyzer, the intrinsic content analyzer being communicatively 
connected to an audio-visual source, the intrinsic content analyzer being adapted to 
search the audio-visual source for intrinsic data and being adapted to extract 
intrinsic data using an extraction algorithm, as (col. 7 lines 62 - 63), "sampling unit 
34 carries out the operation of creating a media sample of the media content played on 
the client media player 14", where the media content is the audio-visual source. 

• an extrinsic content analyzer, the extrinsic content analyzer being communicatively 
connected to an extrinsic information source, the extrinsic content analyzer being 
adapted to search the extrinsic information source and being adapted to retrieve 
extrinsic data using a retrieval algorithm, as (col. 8 lines 26 - 27) "media player 14 
and transmit the sample to the lookup server 12... the lookup server 12 provides the 
information related to the media sample". 

• wherein the intrinsic data and the extrinsic data are correlated, thereby providing a 
multi-source data structure, as (col. 8 lines 30 - 32) "content-related information is 
received by the user interface 38" where the content-related information is the correlation 
between intrinsic and extrinsic data. 

As per Claim 2, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein the retrieval of the extrinsic data is based on the extracted intrinsic data, as 
(col. 8 lines 25 - 30) "media player 14 and transmit the sample to the lookup server 
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12... the lookup server 12 provides the information related to the media sample... content- 
related information is received by the user interface 38". 
As per Claim 3, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein the extraction and/or retrieval algorithm(s) is/are provided by a module, as 
(col. 7 lines 62 - 63), "sampling unit 34 carries out the operation of creating a media 
sample of the media content played on the client media player 14", where the sampling 
unit is the module as claimed. 

As per Claim 4, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein a query is provided by a user, the query being provided to the extraction 
algorithm and wherein the intrinsic data is extracted in accordance with the query, 
as (col. 8 lines 20 - 23) "a user my issue a request for content-related information via the 
user-interface 38. This request is communicated to the sampling unit 34 for further 
processing", where the request is claimed query. 

As per Claim 5, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein a query is provided by a user, the query being provided to the retrieval 
algorithm and wherein the extrinsic data is retrieved in accordance with the query, 
as (col. 8 lines 20 - 30) "a user my issue a request for content-related information via the 
user-interface 38. This request is communicated to the sampling unit 34 for further 
processing.. . transmit the sample to the lookup server 12. . .the lookup server 12 provides 
the information related to the media sample", where the request is claimed query. 

As per Claim 8, Claim 1 is incorporated and further Ikezoye discloses: 
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• wherein the extrinsic information source is connected to and may be accessed via 
the Internet (103), as (col. 3 lines 23 - 24) "the lookup server is generally connected to 
the client media players via an Internet connection." 

As per Claim 9, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein the extrinsic information source is a film screenplay (102) as (col. 4 lines 50 - 
51) "media content source may also be audio CDs, DVD or other formats suitable for 
presentation on the media play devices" where DVD can contain a film screenplay. 

As per Claim 12, Claim 9 is incorporated and further Ikezoye discloses: 

• wherein a feature in a film is analyzed based on information included in the 
screenplay, as (col. 8 lines 26 - 27) "media player 14 and transmit the sample to the 
lookup server 12... the lookup server 12 provides the information related to the media 
sample" and furthermore (col. 4 lines 50-51) "media content source may also be audio 
CDs, DVD or other formats suitable for presentation on the media play devices" where 
DVD can contain a film screenplay. 

As per Claim 17, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein a high- level information structure (5-9) is generated in accordance with the 
multi-source data structure, as (col. 8 lines 30 - 32) "content-related information is 
received by the user interface 38" where the content-related information returned is high- 
level as claimed. 

As per Claim 18, Claim 17 is incorporated and further Ikezoye discloses: 

• wherein the high- level information structure may be stored on a storage medium, as 

(col. 8 lines 30 - 32) "content-related information is received by the user interface 38" 
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where the user interface is in the client device consisting of a storage medium (col. 6 
lines 30 - 35). 

As per Claim 19, Claim 17 is incorporated and further Ikezoye discloses: 

• wherein an update high-level information structure is generated, the updated high- 
level information structure being an already existing high-level information 
structure which is updated in accordance with the multi-source data structure, as 

(col. 8 lines 30 - 32) "content-related information is received by the user interface 38 M 
where the user interface is in the client device consisting of a storage medium (col. 6 
lines 30-35) where each time the content-related information is received it is an update. 
As per Claim 20, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein the retrieval algorithm is a dynamic retrieval algorithm adapted to 
dynamically update itself by including additional functionalities in accordance with 
retrieved extrinsic data, as (col. 8 lines 30 - 32) "content-related information is 
received by the user interface 38" where each time the content-related information is 
received it is displayed to the user and therefore stored as an update. 

As per Claim 21, Claim 20 is incorporated and further Ikezoye discloses: 

• wherein the additional functionalities is obtained by training the retrieval algorithm 
on a set of features from intrinsic data using labels obtained from the extrinsic data, 
(col. 9 lines 3 - 8) discloses a log unit that maintains media request such as media, type, 
genre or category which assist in training the lookup server by keeping track of what is 
requested. 

As per Claim 22, Claim 9 is incorporated and further Ikezoye discloses: 
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• wherein the training is performed using at least one screenplay, as (col. 9 lines 3-8) 
discloses a log unit that maintains media request such as media, type, genre or category 
which assist in training the lookup server by keeping track of what is requested. 
Furthermore (col. 4 lines 50-51) "media content source may also be audio CDs, DVD or 
other formats suitable for presentation on the media play devices" where DVD can 
contain a film screenplay. 

As per Claim 23, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein an automatic ground truth identification in a film is obtained based on the 
multi-source data structure for use in benchmarking algorithms on audio-visual 
content, as (col. 8 lines 47 - 59) "sequentially compares each reference sample in the 
structure 47 to the media sample provided by the media player. . .The reference that has 
the smallest distance to any frame in the sample is considered a match" which shows the 
multi-source data correlated automatically which is automatic ground truth as claimed. 

As per Claim 25, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein an automatic labeling in a film is obtained based on the multi-source data 
structure, as (col. 8 lines 62 - 63) "This content-related information may include such 
information as song title, artist, and album name". 

Claim Rejections - 35 (JSC § 103 
10. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
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having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

.11. Claim 6, 7, 10-11, 13-21, 24-25, 27-28 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ikezoye and in view of Witteman (US 6243676 Bl) 
As per Claim 6, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein a feature reflected in the intrinsic and extrinsic data include textual, audio 
and/or visual features, as (col. 8 lines 60 - 65) discloses the related matching records 
return to the user include audio and or visual feature. 

• But Ikezoye fails to disclose textual features. 

However, Witteman teaches the above limitations as (col. 4 line 34) "closed caption text 
feed is then separated." , 

Therefore it would have been obvious to one of the ordinary skill in the art at the time of 
the invention made to incorporate the teaching of Witteman into the teaching of Ikezoye 
because one of the ordinary skill in the art would have been motivated to use such a modification 
for the purpose of providing additional options and description describing the content so that 
more relevant information can be retrieved. 

As per Claim 7, Claim 1 is incorporated and further Ikezoye discloses: 

• wherein the audio\- visual source is a film (101) as (col. 2 lines 53) "media content, 
such as audio/video played on the media player", where a video played on the media 
player is a film 

• and wherein the extracted data include textual (104), audio and/or visual features 
(105, 106) as (col. 8 lines 60 - 65) discloses the related matching records return to the 
user include audio and or visual feature. 
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• But Ikezoye fails to disclose textual features. 

However, Witteman teaches the above limitations as (col. 4 line 34) "closed caption text 
feed is then separated." 

Therefore it would have been obvious to one of the ordinary skill in the art at the time of 
the invention made to incorporate the teaching of Witteman into the teaching of Ikezoye 
because one of the ordinary skill in the art would have been motivated to use such a modification 
for the purpose of providing additional options and description describing the content so that 
more relevant information can be retrieved. 

As per Claim 10, Claim 9 is incorporated and further Ikezoye discloses: 

• extrinsic data is retrieved based on information extracted from audio video source, 

as (col. 8 lines 26 - 27) "media player 14 and transmit the sample to the lookup server 
12.. .the lookup server 12 provides the information related to the media sample". 

• But Ikezoye does not disclose wherein the extrinsic content analyzer include 
knowledge about screenplay grammar, and wherein the extrinsic data is retrieved 
based on information extracted from the screenplay by use of the screenplay grammar 
However, Witteman teaches the extrinsic content analyzer include knowledge about the 

screen play grammar as (col. 4 lines 34 - 39) "text feed is compared to a dictionary of key words 
and phrases (step 416)" 

Therefore it would have been obvious to one of the ordinary skill in the art at the time of 
the invention made to incorporate the teaching of Witteman into the teaching of Ikezoye 
because one of the ordinary skill in the art would have been motivated to use such a modification 
for the purpose of providing an analyzer that understands the extracted text so that it can be 
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searched by the lookup server in Ikezoye' s system which will provide information related to the 
source. Therefor providing additional options to describe the content for retrieving more 
relevant information. 

As per Claim 11, Claim 9 is incorporated and further Ikezoye does not disclose: 

• wherein the identification (5) of persons in a film is obtained by means of the 
screenplay. 

• However Witteman discloses identifying artist as content related information on audio 
CDs (col. 4 lines 34- 35) and furthermore discloses DVDs (col. 4 lines 50-51) "media 
content source may also be .DVD" which can contain a screenplay. 

Therefore it would have been obvious to one of the ordinary skill in the art at the time of 
the invention made to incorporate the teaching of Witteman into the teaching of Ikezoye 
because one of the ordinary skill in the art would have been motivated to use such a modification 
for the purpose of providing an analyzer that understands the extracted text so that it can be 
searched by the lookup server in Ikezoye' s system which will provide information related to the 
source. Therefor providing additional options to describe the content for retrieving more 
relevant information. 

As per Claim 13, Claim 1 is incorporated and further Ikezoye does not disclose: 

• wherein the correlation of the intrinsic and extrinsic data is time correlation 
(121), thereby providing a multisource data structure where a feature reflected in 
the intrinsic data is time correlated to a feature reflected in the extrinsic data. 
However Witteman discloses the above limitation as (FIG. 3) which discloses the 

linking the extracted text and related searches bases on time. 
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Therefore it would have been obvious to one of the ordinary skill in the art at the time of 
the invention made to incorporate the teaching of Witteman into the teaching of Ikezoye 
because one of the ordinary skill in the art would have been motivated to use such a modification 
for the purpose of providing a related content within similar a category of time which provides a 
data package consisting of multiple information related to the source. 
As per Claim 14, Claim 13 is incorporated and further Ikezoye does not disclose: 

• wherein the time correlation is obtained by an alignment of a dialogue (120) in the 
screenplay to the spoken text (104) in the film and thereby providing a timestamped 
transcript (121) of the Him. 

However Witteman discloses the above limitation as (FIG. 3) which discloses the 
linking the extracted text and related searches bases on time recognized speech. 

Therefore it would have been obvious to one of the ordinary skill in the art at the time of 
the invention made to incorporate the teaching of Witteman into the teaching of Ikezoye 
because one of the ordinary skill in the art would have been motivated to use such a modification 
for the purpose of providing a related content within similar a category of time which provides a 
data package consisting of multiple information related to the source. 
As per Claim 15, Claim 14 is incorporated and further Ikezoye does not disclose: 

• wherein a speaker identification in the film is obtained from the time stamped 
transcript. 

However Witteman discloses the above limitation as (col. 4 lines 42 - 43) "process 400 
then determines a start of the audio block, indexes the audio block and sends the audio block to 
an information store" where determining the start of the audio block is claimed time stamp. 
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Therefore it would have been obvious to one of the ordinary skill in the art at the time of 
the invention made to incorporate the teaching of Witteman into the teaching of Ikezoye 
because one of the ordinary skill in the art would have been motivated to use such a modification 
for the purpose of providing a related content within similar a category of time which provides a 
data package consisting of multiple information related to the source. 
As per Claim 16, Claim 9 is incorporated and further Ikezoye does not disclose: 

• wherein the screenplay is compared with the spoken text in the film by means of a 
self-similarity matrix (30). 

However Witteman discloses the above limitation as (FIG. 3) which discloses the 
linking the extracted text and related searches of the audio based on time and recognized speech. 

Therefore it would have been obvious to one of the ordinary skill in the art at the time of 
the invention made to incorporate the teaching of Witteman into the teaching of Ikezoye 
because one of the ordinary skill in the art would have been motivated to use such a modification 
for the purpose of providing a related content within similar a category of time which provides a 
data package consisting of multiple information related to the source. 
As per Claim 24, Claim 1 is incorporated and further Ikezoye does not disclose: 

• wherein an automatic scene content understanding in a film is obtained based on the 
textual description in the screenplay and the audio-visual features from the film 
content. 

However Witteman discloses the above limitation as (FIG. 3) which discloses the 
linking the extracted text and related searches of the audio based on time and recognized speech. 
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Therefore it would have been obvious to one of the ordinary skill in the art at the time of 
the invention made to incorporate the teaching of Witteman into the teaching of Ikezoye 
because one of the ordinary skill in the art would have been motivated to use such a modification 
for the purpose of providing a related content within similar a category of time which provides a 
data package consisting of multiple information related to the source. 

Claims 26, 27, 28, 29 are method claims for integrative analyses of intrinsic and extrinsic audio- 
visual source, corresponding to the method claims 1, 17, 10, 20 respectively, and are rejected 
under the same reason set forth in connection to rejections of claims 1, 17, 10, 20 respectively 
above. 

Conclusion 

12. The prior art made or recorded and not relied upon is considered pertinent to applicant's 
disclosure. 

TITLE: System and method for media search and playback, US 20040167890 Al 

TITLE: Simplified searching for media services using a control device, US 20040177063 Al 

TITLE: Method and apparatus for searching recommended music in the internet, and a 

computer-readable medium encoded with a plurality of processor-executable instruction 

sequences for searching recommended music in the internet, US 20040193649 Al 

TITLE: Establishing and interacting with on-line media collections using identifiers in media 

signals, US 6829368.B2 

TITLE: Digital media recognition apparatus and methods, US 20040133927 Al 

TITLE: Multimedia computer system with story segmentation capability and operating program 

therefor including finite automation video parser, US 6363380 Bl 
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13. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Dennis Truong whose telephone number is (571) 270-3157. The 
examiner can normally be reached on MON - FRI: 7:30 - 5:00 PM EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, MOHAMMAD Ali can be reached on (571) 272-4105. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 




Dennis Truong 
Patent Examiner 



